{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "\n",
    "<a href='https://www.udemy.com/user/joseportilla/'><img src='../Pierian_Data_Logo.png'/></a>\n",
    "___\n",
    "<center><em>Content Copyright by Pierian Data</em></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Scraping Exercises - Solutions\n",
    "\n",
    "## Complete the Tasks Below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TASK: Import any libraries you think you'll need to scrape a website.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import bs4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TASK: Use requests library and BeautifulSoup to connect to http://quotes.toscrape.com/ and get the HMTL text from the homepage.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "res = requests.get(\"http://quotes.toscrape.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!DOCTYPE html>\\n<html lang=\"en\">\\n<head>\\n\\t<meta charset=\"UTF-8\">\\n\\t<title>Quotes to Scrape</title>\\n    <link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\">\\n    <link rel=\"stylesheet\" href=\"/static/main.css\">\\n</head>\\n<body>\\n    <div class=\"container\">\\n        <div class=\"row header-box\">\\n            <div class=\"col-md-8\">\\n                <h1>\\n                    <a href=\"/\" style=\"text-decoration: none\">Quotes to Scrape</a>\\n                </h1>\\n            </div>\\n            <div class=\"col-md-4\">\\n                <p>\\n                \\n                    <a href=\"/login\">Login</a>\\n                \\n                </p>\\n            </div>\\n        </div>\\n    \\n\\n<div class=\"row\">\\n    <div class=\"col-md-8\">\\n\\n    <div class=\"quote\" itemscope itemtype=\"http://schema.org/CreativeWork\">\\n        <span class=\"text\" itemprop=\"text\">“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”</span>\\n        <span>by <small class=\"author\" itemprop=\"author\">Albert Einstein</small>\\n        <a href=\"/author/Albert-Einstein\">(about)</a>\\n        </span>\\n        <div class=\"tags\">\\n            Tags:\\n            <meta class=\"keywords\" itemprop=\"keywords\" content=\"change,deep-thoughts,thinking,world\" /    > \\n            \\n            <a class=\"tag\" href=\"/tag/change/page/1/\">change</a>\\n            \\n            <a class=\"tag\" href=\"/tag/deep-thoughts/page/1/\">deep-thoughts</a>\\n            \\n            <a class=\"tag\" href=\"/tag/thinking/page/1/\">thinking</a>\\n            \\n            <a class=\"tag\" href=\"/tag/world/page/1/\">world</a>\\n            \\n        </div>\\n    </div>\\n\\n    <div class=\"quote\" itemscope itemtype=\"http://schema.org/CreativeWork\">\\n        <span class=\"text\" itemprop=\"text\">“It is our choices, Harry, that show what we truly are, far more than our abilities.”</span>\\n        <span>by <small class=\"author\" itemprop=\"author\">J.K. Rowling</small>\\n        <a href=\"/author/J-K-Rowling\">(about)</a>\\n        </span>\\n        <div class=\"tags\">\\n            Tags:\\n            <meta class=\"keywords\" itemprop=\"keywords\" content=\"abilities,choices\" /    > \\n            \\n            <a class=\"tag\" href=\"/tag/abilities/page/1/\">abilities</a>\\n            \\n            <a class=\"tag\" href=\"/tag/choices/page/1/\">choices</a>\\n            \\n        </div>\\n    </div>\\n\\n    <div class=\"quote\" itemscope itemtype=\"http://schema.org/CreativeWork\">\\n        <span class=\"text\" itemprop=\"text\">“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”</span>\\n        <span>by <small class=\"author\" itemprop=\"author\">Albert Einstein</small>\\n        <a href=\"/author/Albert-Einstein\">(about)</a>\\n        </span>\\n        <div class=\"tags\">\\n            Tags:\\n            <meta class=\"keywords\" itemprop=\"keywords\" content=\"inspirational,life,live,miracle,miracles\" /    > \\n            \\n            <a class=\"tag\" href=\"/tag/inspirational/page/1/\">inspirational</a>\\n            \\n            <a class=\"tag\" href=\"/tag/life/page/1/\">life</a>\\n            \\n            <a class=\"tag\" href=\"/tag/live/page/1/\">live</a>\\n            \\n            <a class=\"tag\" href=\"/tag/miracle/page/1/\">miracle</a>\\n            \\n            <a class=\"tag\" href=\"/tag/miracles/page/1/\">miracles</a>\\n            \\n        </div>\\n    </div>\\n\\n    <div class=\"quote\" itemscope itemtype=\"http://schema.org/CreativeWork\">\\n        <span class=\"text\" itemprop=\"text\">“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”</span>\\n        <span>by <small class=\"author\" itemprop=\"author\">Jane Austen</small>\\n        <a href=\"/author/Jane-Austen\">(about)</a>\\n        </span>\\n        <div class=\"tags\">\\n            Tags:\\n            <meta class=\"keywords\" itemprop=\"keywords\" content=\"aliteracy,books,classic,humor\" /    > \\n            \\n            <a class=\"tag\" href=\"/tag/aliteracy/page/1/\">aliteracy</a>\\n            \\n            <a class=\"tag\" href=\"/tag/books/page/1/\">books</a>\\n            \\n            <a class=\"tag\" href=\"/tag/classic/page/1/\">classic</a>\\n            \\n            <a class=\"tag\" href=\"/tag/humor/page/1/\">humor</a>\\n            \\n        </div>\\n    </div>\\n\\n    <div class=\"quote\" itemscope itemtype=\"http://schema.org/CreativeWork\">\\n        <span class=\"text\" itemprop=\"text\">“Imperfection is beauty, madness is genius and it&#39;s better to be absolutely ridiculous than absolutely boring.”</span>\\n        <span>by <small class=\"author\" itemprop=\"author\">Marilyn Monroe</small>\\n        <a href=\"/author/Marilyn-Monroe\">(about)</a>\\n        </span>\\n        <div class=\"tags\">\\n            Tags:\\n            <meta class=\"keywords\" itemprop=\"keywords\" content=\"be-yourself,inspirational\" /    > \\n            \\n            <a class=\"tag\" href=\"/tag/be-yourself/page/1/\">be-yourself</a>\\n            \\n            <a class=\"tag\" href=\"/tag/inspirational/page/1/\">inspirational</a>\\n            \\n        </div>\\n    </div>\\n\\n    <div class=\"quote\" itemscope itemtype=\"http://schema.org/CreativeWork\">\\n        <span class=\"text\" itemprop=\"text\">“Try not to become a man of success. Rather become a man of value.”</span>\\n        <span>by <small class=\"author\" itemprop=\"author\">Albert Einstein</small>\\n        <a href=\"/author/Albert-Einstein\">(about)</a>\\n        </span>\\n        <div class=\"tags\">\\n            Tags:\\n            <meta class=\"keywords\" itemprop=\"keywords\" content=\"adulthood,success,value\" /    > \\n            \\n            <a class=\"tag\" href=\"/tag/adulthood/page/1/\">adulthood</a>\\n            \\n            <a class=\"tag\" href=\"/tag/success/page/1/\">success</a>\\n            \\n            <a class=\"tag\" href=\"/tag/value/page/1/\">value</a>\\n            \\n        </div>\\n    </div>\\n\\n    <div class=\"quote\" itemscope itemtype=\"http://schema.org/CreativeWork\">\\n        <span class=\"text\" itemprop=\"text\">“It is better to be hated for what you are than to be loved for what you are not.”</span>\\n        <span>by <small class=\"author\" itemprop=\"author\">André Gide</small>\\n        <a href=\"/author/Andre-Gide\">(about)</a>\\n        </span>\\n        <div class=\"tags\">\\n            Tags:\\n            <meta class=\"keywords\" itemprop=\"keywords\" content=\"life,love\" /    > \\n            \\n            <a class=\"tag\" href=\"/tag/life/page/1/\">life</a>\\n            \\n            <a class=\"tag\" href=\"/tag/love/page/1/\">love</a>\\n            \\n        </div>\\n    </div>\\n\\n    <div class=\"quote\" itemscope itemtype=\"http://schema.org/CreativeWork\">\\n        <span class=\"text\" itemprop=\"text\">“I have not failed. I&#39;ve just found 10,000 ways that won&#39;t work.”</span>\\n        <span>by <small class=\"author\" itemprop=\"author\">Thomas A. Edison</small>\\n        <a href=\"/author/Thomas-A-Edison\">(about)</a>\\n        </span>\\n        <div class=\"tags\">\\n            Tags:\\n            <meta class=\"keywords\" itemprop=\"keywords\" content=\"edison,failure,inspirational,paraphrased\" /    > \\n            \\n            <a class=\"tag\" href=\"/tag/edison/page/1/\">edison</a>\\n            \\n            <a class=\"tag\" href=\"/tag/failure/page/1/\">failure</a>\\n            \\n            <a class=\"tag\" href=\"/tag/inspirational/page/1/\">inspirational</a>\\n            \\n            <a class=\"tag\" href=\"/tag/paraphrased/page/1/\">paraphrased</a>\\n            \\n        </div>\\n    </div>\\n\\n    <div class=\"quote\" itemscope itemtype=\"http://schema.org/CreativeWork\">\\n        <span class=\"text\" itemprop=\"text\">“A woman is like a tea bag; you never know how strong it is until it&#39;s in hot water.”</span>\\n        <span>by <small class=\"author\" itemprop=\"author\">Eleanor Roosevelt</small>\\n        <a href=\"/author/Eleanor-Roosevelt\">(about)</a>\\n        </span>\\n        <div class=\"tags\">\\n            Tags:\\n            <meta class=\"keywords\" itemprop=\"keywords\" content=\"misattributed-eleanor-roosevelt\" /    > \\n            \\n            <a class=\"tag\" href=\"/tag/misattributed-eleanor-roosevelt/page/1/\">misattributed-eleanor-roosevelt</a>\\n            \\n        </div>\\n    </div>\\n\\n    <div class=\"quote\" itemscope itemtype=\"http://schema.org/CreativeWork\">\\n        <span class=\"text\" itemprop=\"text\">“A day without sunshine is like, you know, night.”</span>\\n        <span>by <small class=\"author\" itemprop=\"author\">Steve Martin</small>\\n        <a href=\"/author/Steve-Martin\">(about)</a>\\n        </span>\\n        <div class=\"tags\">\\n            Tags:\\n            <meta class=\"keywords\" itemprop=\"keywords\" content=\"humor,obvious,simile\" /    > \\n            \\n            <a class=\"tag\" href=\"/tag/humor/page/1/\">humor</a>\\n            \\n            <a class=\"tag\" href=\"/tag/obvious/page/1/\">obvious</a>\\n            \\n            <a class=\"tag\" href=\"/tag/simile/page/1/\">simile</a>\\n            \\n        </div>\\n    </div>\\n\\n    <nav>\\n        <ul class=\"pager\">\\n            \\n            \\n            <li class=\"next\">\\n                <a href=\"/page/2/\">Next <span aria-hidden=\"true\">&rarr;</span></a>\\n            </li>\\n            \\n        </ul>\\n    </nav>\\n    </div>\\n    <div class=\"col-md-4 tags-box\">\\n        \\n            <h2>Top Ten tags</h2>\\n            \\n            <span class=\"tag-item\">\\n            <a class=\"tag\" style=\"font-size: 28px\" href=\"/tag/love/\">love</a>\\n            </span>\\n            \\n            <span class=\"tag-item\">\\n            <a class=\"tag\" style=\"font-size: 26px\" href=\"/tag/inspirational/\">inspirational</a>\\n            </span>\\n            \\n            <span class=\"tag-item\">\\n            <a class=\"tag\" style=\"font-size: 26px\" href=\"/tag/life/\">life</a>\\n            </span>\\n            \\n            <span class=\"tag-item\">\\n            <a class=\"tag\" style=\"font-size: 24px\" href=\"/tag/humor/\">humor</a>\\n            </span>\\n            \\n            <span class=\"tag-item\">\\n            <a class=\"tag\" style=\"font-size: 22px\" href=\"/tag/books/\">books</a>\\n            </span>\\n            \\n            <span class=\"tag-item\">\\n            <a class=\"tag\" style=\"font-size: 14px\" href=\"/tag/reading/\">reading</a>\\n            </span>\\n            \\n            <span class=\"tag-item\">\\n            <a class=\"tag\" style=\"font-size: 10px\" href=\"/tag/friendship/\">friendship</a>\\n            </span>\\n            \\n            <span class=\"tag-item\">\\n            <a class=\"tag\" style=\"font-size: 8px\" href=\"/tag/friends/\">friends</a>\\n            </span>\\n            \\n            <span class=\"tag-item\">\\n            <a class=\"tag\" style=\"font-size: 8px\" href=\"/tag/truth/\">truth</a>\\n            </span>\\n            \\n            <span class=\"tag-item\">\\n            <a class=\"tag\" style=\"font-size: 6px\" href=\"/tag/simile/\">simile</a>\\n            </span>\\n            \\n        \\n    </div>\\n</div>\\n\\n    </div>\\n    <footer class=\"footer\">\\n        <div class=\"container\">\\n            <p class=\"text-muted\">\\n                Quotes by: <a href=\"https://www.goodreads.com/quotes\">GoodReads.com</a>\\n            </p>\\n            <p class=\"copyright\">\\n                Made with <span class=\\'sh-red\\'>❤</span> by <a href=\"https://scrapinghub.com\">Scrapinghub</a>\\n            </p>\\n        </div>\\n    </footer>\\n</body>\\n</html>'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TASK: Get the names of all the authors on the first page.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "soup = bs4.BeautifulSoup(res.text,'lxml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<small class=\"author\" itemprop=\"author\">Albert Einstein</small>,\n",
       " <small class=\"author\" itemprop=\"author\">J.K. Rowling</small>,\n",
       " <small class=\"author\" itemprop=\"author\">Albert Einstein</small>,\n",
       " <small class=\"author\" itemprop=\"author\">Jane Austen</small>,\n",
       " <small class=\"author\" itemprop=\"author\">Marilyn Monroe</small>,\n",
       " <small class=\"author\" itemprop=\"author\">Albert Einstein</small>,\n",
       " <small class=\"author\" itemprop=\"author\">André Gide</small>,\n",
       " <small class=\"author\" itemprop=\"author\">Thomas A. Edison</small>,\n",
       " <small class=\"author\" itemprop=\"author\">Eleanor Roosevelt</small>,\n",
       " <small class=\"author\" itemprop=\"author\">Steve Martin</small>]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select(\".author\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# I used a set to not worry about repeat authors.\n",
    "authors = set() \n",
    "for name in soup.select(\".author\"):\n",
    "    authors.add(name.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Albert Einstein',\n",
       " 'André Gide',\n",
       " 'Eleanor Roosevelt',\n",
       " 'J.K. Rowling',\n",
       " 'Jane Austen',\n",
       " 'Marilyn Monroe',\n",
       " 'Steve Martin',\n",
       " 'Thomas A. Edison'}"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "authors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TASK: Create a list of all the quotes on the first page.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "quotes = []\n",
    "for quote in soup.select(\".text\"):\n",
    "    quotes.append(quote.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',\n",
       " '“It is our choices, Harry, that show what we truly are, far more than our abilities.”',\n",
       " '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”',\n",
       " '“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”',\n",
       " \"“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”\",\n",
       " '“Try not to become a man of success. Rather become a man of value.”',\n",
       " '“It is better to be hated for what you are than to be loved for what you are not.”',\n",
       " \"“I have not failed. I've just found 10,000 ways that won't work.”\",\n",
       " \"“A woman is like a tea bag; you never know how strong it is until it's in hot water.”\",\n",
       " '“A day without sunshine is like, you know, night.”']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quotes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TASK: Inspect the site and use Beautiful Soup to extract the top ten tags from the requests text shown on the top right from the home page (e.g Love,Inspirational,Life, etc...). HINT: Keep in mind there are also tags underneath each quote, try to find a class only present in the top right tags, perhaps check the span.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "soup = bs4.BeautifulSoup(res.text,'lxml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<span class=\"tag-item\">\n",
       " <a class=\"tag\" href=\"/tag/love/\" style=\"font-size: 28px\">love</a>\n",
       " </span>, <span class=\"tag-item\">\n",
       " <a class=\"tag\" href=\"/tag/inspirational/\" style=\"font-size: 26px\">inspirational</a>\n",
       " </span>, <span class=\"tag-item\">\n",
       " <a class=\"tag\" href=\"/tag/life/\" style=\"font-size: 26px\">life</a>\n",
       " </span>, <span class=\"tag-item\">\n",
       " <a class=\"tag\" href=\"/tag/humor/\" style=\"font-size: 24px\">humor</a>\n",
       " </span>, <span class=\"tag-item\">\n",
       " <a class=\"tag\" href=\"/tag/books/\" style=\"font-size: 22px\">books</a>\n",
       " </span>, <span class=\"tag-item\">\n",
       " <a class=\"tag\" href=\"/tag/reading/\" style=\"font-size: 14px\">reading</a>\n",
       " </span>, <span class=\"tag-item\">\n",
       " <a class=\"tag\" href=\"/tag/friendship/\" style=\"font-size: 10px\">friendship</a>\n",
       " </span>, <span class=\"tag-item\">\n",
       " <a class=\"tag\" href=\"/tag/friends/\" style=\"font-size: 8px\">friends</a>\n",
       " </span>, <span class=\"tag-item\">\n",
       " <a class=\"tag\" href=\"/tag/truth/\" style=\"font-size: 8px\">truth</a>\n",
       " </span>, <span class=\"tag-item\">\n",
       " <a class=\"tag\" href=\"/tag/simile/\" style=\"font-size: 6px\">simile</a>\n",
       " </span>]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select('.tag-item')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "love\n",
      "\n",
      "\n",
      "inspirational\n",
      "\n",
      "\n",
      "life\n",
      "\n",
      "\n",
      "humor\n",
      "\n",
      "\n",
      "books\n",
      "\n",
      "\n",
      "reading\n",
      "\n",
      "\n",
      "friendship\n",
      "\n",
      "\n",
      "friends\n",
      "\n",
      "\n",
      "truth\n",
      "\n",
      "\n",
      "simile\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for item in soup.select(\".tag-item\"):\n",
    "    print(item.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TASK: Notice how there is more than one page, and subsequent pages look like this http://quotes.toscrape.com/page/2/. Use what you know about for loops and string concatenation to loop through all the pages and get all the unique authors on the website. Keep in mind there are many ways to achieve this, also note that you will need to somehow figure out how to check that your loop is on the last page with quotes. For debugging purposes, I will let you know that there are only 10 pages, so the last page is http://quotes.toscrape.com/page/10/, but try to create a loop that is robust enough that it wouldn't matter to know the amount of pages beforehand, perhaps use try/except for this, its up to you!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Possible Solution #1 ( Assuming You Know Number of Pages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "url = 'http://quotes.toscrape.com/page/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "authors = set()\n",
    "\n",
    "for page in range(1,10):\n",
    "\n",
    "    # Concatenate to get new page URL\n",
    "    page_url = url+str(page)\n",
    "    # Obtain Request\n",
    "    res = requests.get(page_url)\n",
    "    # Turn into Soup\n",
    "    soup = bs4.BeautifulSoup(res.text,'lxml')\n",
    "    # Add Authors to our set\n",
    "    for name in soup.select(\".author\"):\n",
    "        authors.add(name.text)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Possible Solution #2 ( Unknown Number of Pages, but knowledge of last page)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check what the last invalid page looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Choose some huge page number we know doesn't exist\n",
    "page_url = url+str(9999999)\n",
    "\n",
    "# Obtain Request\n",
    "res = requests.get(page_url)\n",
    "\n",
    "# Turn into Soup\n",
    "soup = bs4.BeautifulSoup(res.text,'lxml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<!DOCTYPE html>\n",
       "<html lang=\"en\">\n",
       "<head>\n",
       "<meta charset=\"utf-8\"/>\n",
       "<title>Quotes to Scrape</title>\n",
       "<link href=\"/static/bootstrap.min.css\" rel=\"stylesheet\"/>\n",
       "<link href=\"/static/main.css\" rel=\"stylesheet\"/>\n",
       "</head>\n",
       "<body>\n",
       "<div class=\"container\">\n",
       "<div class=\"row header-box\">\n",
       "<div class=\"col-md-8\">\n",
       "<h1>\n",
       "<a href=\"/\" style=\"text-decoration: none\">Quotes to Scrape</a>\n",
       "</h1>\n",
       "</div>\n",
       "<div class=\"col-md-4\">\n",
       "<p>\n",
       "<a href=\"/login\">Login</a>\n",
       "</p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"row\">\n",
       "<div class=\"col-md-8\">\n",
       "\n",
       "No quotes found!\n",
       "\n",
       "    <nav>\n",
       "<ul class=\"pager\">\n",
       "<li class=\"previous\">\n",
       "<a href=\"/page/9999998/\"><span aria-hidden=\"true\">←</span> Previous</a>\n",
       "</li>\n",
       "</ul>\n",
       "</nav>\n",
       "</div>\n",
       "<div class=\"col-md-4 tags-box\">\n",
       "<h2>Top Ten tags</h2>\n",
       "<span class=\"tag-item\">\n",
       "<a class=\"tag\" href=\"/tag/love/\" style=\"font-size: 28px\">love</a>\n",
       "</span>\n",
       "<span class=\"tag-item\">\n",
       "<a class=\"tag\" href=\"/tag/inspirational/\" style=\"font-size: 26px\">inspirational</a>\n",
       "</span>\n",
       "<span class=\"tag-item\">\n",
       "<a class=\"tag\" href=\"/tag/life/\" style=\"font-size: 26px\">life</a>\n",
       "</span>\n",
       "<span class=\"tag-item\">\n",
       "<a class=\"tag\" href=\"/tag/humor/\" style=\"font-size: 24px\">humor</a>\n",
       "</span>\n",
       "<span class=\"tag-item\">\n",
       "<a class=\"tag\" href=\"/tag/books/\" style=\"font-size: 22px\">books</a>\n",
       "</span>\n",
       "<span class=\"tag-item\">\n",
       "<a class=\"tag\" href=\"/tag/reading/\" style=\"font-size: 14px\">reading</a>\n",
       "</span>\n",
       "<span class=\"tag-item\">\n",
       "<a class=\"tag\" href=\"/tag/friendship/\" style=\"font-size: 10px\">friendship</a>\n",
       "</span>\n",
       "<span class=\"tag-item\">\n",
       "<a class=\"tag\" href=\"/tag/friends/\" style=\"font-size: 8px\">friends</a>\n",
       "</span>\n",
       "<span class=\"tag-item\">\n",
       "<a class=\"tag\" href=\"/tag/truth/\" style=\"font-size: 8px\">truth</a>\n",
       "</span>\n",
       "<span class=\"tag-item\">\n",
       "<a class=\"tag\" href=\"/tag/simile/\" style=\"font-size: 6px\">simile</a>\n",
       "</span>\n",
       "</div>\n",
       "</div>\n",
       "</div>\n",
       "<footer class=\"footer\">\n",
       "<div class=\"container\">\n",
       "<p class=\"text-muted\">\n",
       "                Quotes by: <a href=\"https://www.goodreads.com/quotes\">GoodReads.com</a>\n",
       "</p>\n",
       "<p class=\"copyright\">\n",
       "                Made with <span class=\"sh-red\">❤</span> by <a href=\"https://scrapinghub.com\">Scrapinghub</a>\n",
       "</p>\n",
       "</div>\n",
       "</footer>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This solution requires that the string \"No quotes found!\" only occurs on the last page.\n",
    "# If for some reason this string was on the other pages, we would need to be more detailed.\n",
    "\"No quotes found!\" in res.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "page_still_valid = True\n",
    "authors = set()\n",
    "page = 1\n",
    "\n",
    "while page_still_valid:\n",
    "\n",
    "    # Concatenate to get new page URL\n",
    "    page_url = url+str(page)\n",
    "    \n",
    "    # Obtain Request\n",
    "    res = requests.get(page_url)\n",
    "    \n",
    "    # Check to see if we're on the last page\n",
    "    if \"No quotes found!\" in res.text:\n",
    "        break\n",
    "    \n",
    "    # Turn into Soup\n",
    "    soup = bs4.BeautifulSoup(res.text,'lxml')\n",
    "    \n",
    "    # Add Authors to our set\n",
    "    for name in soup.select(\".author\"):\n",
    "        authors.add(name.text)\n",
    "        \n",
    "    # Go to Next Page\n",
    "    page += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Albert Einstein',\n",
       " 'Alexandre Dumas fils',\n",
       " 'Alfred Tennyson',\n",
       " 'Allen Saunders',\n",
       " 'André Gide',\n",
       " 'Ayn Rand',\n",
       " 'Bob Marley',\n",
       " 'C.S. Lewis',\n",
       " 'Charles Bukowski',\n",
       " 'Charles M. Schulz',\n",
       " 'Douglas Adams',\n",
       " 'Dr. Seuss',\n",
       " 'E.E. Cummings',\n",
       " 'Eleanor Roosevelt',\n",
       " 'Elie Wiesel',\n",
       " 'Ernest Hemingway',\n",
       " 'Friedrich Nietzsche',\n",
       " 'Garrison Keillor',\n",
       " 'George Bernard Shaw',\n",
       " 'George Carlin',\n",
       " 'George Eliot',\n",
       " 'George R.R. Martin',\n",
       " 'Harper Lee',\n",
       " 'Haruki Murakami',\n",
       " 'Helen Keller',\n",
       " 'J.D. Salinger',\n",
       " 'J.K. Rowling',\n",
       " 'J.M. Barrie',\n",
       " 'J.R.R. Tolkien',\n",
       " 'James Baldwin',\n",
       " 'Jane Austen',\n",
       " 'Jim Henson',\n",
       " 'Jimi Hendrix',\n",
       " 'John Lennon',\n",
       " 'Jorge Luis Borges',\n",
       " 'Khaled Hosseini',\n",
       " \"Madeleine L'Engle\",\n",
       " 'Marilyn Monroe',\n",
       " 'Mark Twain',\n",
       " 'Martin Luther King Jr.',\n",
       " 'Mother Teresa',\n",
       " 'Pablo Neruda',\n",
       " 'Ralph Waldo Emerson',\n",
       " 'Stephenie Meyer',\n",
       " 'Steve Martin',\n",
       " 'Suzanne Collins',\n",
       " 'Terry Pratchett',\n",
       " 'Thomas A. Edison',\n",
       " 'W.C. Fields',\n",
       " 'William Nicholson'}"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "authors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are lots of other potential solutions that are even more robust and flexible, the main idea is the same though, use a while loop to cycle through potential pages and have a break condition based on the invalid page."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
